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Executive Summary 

In January 2019, the Internal Revenue Service (IRS) Free File program (“Free File” or 
“the program”) opened for its 17 th filing season. Established in 2002, the IRS Free File program 
makes free online tax preparation and electronic filing available to 70 percent of the taxpaying 
population; for Filing Season (FS) 2019, this equated to taxpayers with an adjusted gross income 
(AGI) of $66,000 or less. The program is made possible through a partnership between the IRS 
and the tax software industry. The partnership is governed through a mutually negotiated 
Memorandum of Understanding (MOU) between the IRS and the Free File Alliance (FFA) 1 —a 
coalition of individual private-sector tax preparation companies 2 —that has evolved over the 
program’s lifespan to address issues as they have arisen. 

While cumulatively the program has resulted in more than 53 million free returns e-filed and an 
estimated $1.6 billion in savings to taxpayers 3 , the Free File program has also faced criticism in 
recent years as questions have been raised by Congress and the press about low annual 
participation rates and assertions of inadequate oversight, deceptive business practices, and 
misaligned roles for government and industry. This year’s launch came amid criticisms of the 
program and speculation that industry partners were not acting in the best interest of taxpayers. 

In response to these assertions as well as recent oversight activities and congressional inquiries, 
IRS leadership determined that an independent assessment was necessary to ensure the continued 
operations and integrity of the IRS Free File program. Because the MITRE Corporation operates 
the Service’s federally funded research and development center (FFRDC), the IRS engaged 
MITRE to conduct an independent and objective assessment of the IRS Free File program in its 
current state, providing recommendations for improvements to the program, and by extension to 
the MOU that governs the program. 

Methods 

Beginning in June 2019, MITRE established several workstreams for addressing each of the 
targeted assessment areas reflected in Figure ES-1. Taken together, these workstreams produced 
findings and recommendations related to Free File member compliance with the MOU and 
potential enhancements to the program. Highlights of the assessment approach include: 


1 In 2014, the Free File Alliance officially changed the name of the organization to Free File, Inc. (FFI) doing business as the 
Free File Alliance. While FFI is the listed name on all agreements and MOUs beginning in 2014, the two names are used 
interchangeably. The IRS references the Alliance on IRS.gov “Free File: About the Free File Alliance,” and the organization has 
not changed the name of its own website: https://freefilealliance.org/. This document will use the term Free File Alliance, the 
Alliance, or FFA except when FFI is used in a specific context or quote. 

2 Industry participation in the Free File program is voluntary. As stipulated in the MOU, there is a process for applying for 
membership to the Free File Alliance and companies may leave the program at their discretion. Also outlined in the MOU, IRS 
and FFA leadership can remove members from the program if there are certain MOU violations. Through FS 2019, the Free File 
Alliance included 12 member companies; currently there are only 11 participating members of the program. As of July 2019, 
Drake [1040.com] chose to withdraw from the program and will not be a part of it going forward. Since they were a part of the 
program for FS 2019, they are still included in this assessment. The decision to leave came from the member itself and to 
MITRE’s knowledge were not related to any MOU infractions. 

3 Internal Revenue Service, "IRS, Free File Alliance announce changes to improve program; Improved taxpayer options available 
for 2019 Free File program," Internal Revenue Service, 2 November 2018. [Online], Available: 

https://www.irs.gov/newsroom/irs-ffee-file-alliance-announce-changes-to-improve-program-improved-taxpayer-options- 
available-for-2019-free-file-program. [Accessed 2 July 2019]. 
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• Current Environmental Scan. MITRE conducted an environmental scan to better 
understand the Free File program in terms of historical context, taxpayer participation, the 
role of IRS and industry in its public-private partnership (PPP), the economics of “free” as 
it pertains to Free File, and recent interest in the program. 

• Free File Program Analysis. MITRE conducted stakeholder interviews—including IRS 
leadership—and performed document reviews to assess the IRS Free File program’s 
alignment to objectives, performance, and the IRS management and oversight functions. 

• FFA Member Organization Interviews (“site visits”). MITRE conducted interviews 
with representatives from each Free File member organization to collect information on 
their interactions and agreements with the IRS regarding the Free File program and 
applicable business processes, practices, and techniques for how each Free File member 
detennines if a taxpayer qualifies for Free File. Each of the interviews followed a similar 
approach: standard interview protocols, targeted questions (both related to the IRS Free 
File program as well as the MOU Article 4, Standards of Practice) and in-person or phone 
interviews with participants from each Free File member organizations (selected by the 
member company’s leadership). 

• Web Forensics Analyses. MITRE performed an analysis to detennine if any coding 
within the member’s website influences search engine routing. In addition, MITRE 
analyzed web search query traffic metrics for internet searches that signaled an intent to 
find free tax filing services, based on a representative sample of organic (unpaid) search 
keywords for free tax filing. This included a click-through analysis ofIRS.gov site usage 
metrics to investigate where taxpayers went from the IRS Free File web portal. This 
analysis was also used to detennine how often taxpayers searched for free tax filing 
services (especially during tax filing season) using two primary search engines, Google 
and Bing; to what site(s) these users went following their searches; and the differences in 
search engine results. 

• Taxpayer Journey. MITRE conducted an expert review of the IRS Free File portal as 
well as each individual Free File member offer. This review was twofold in its objective: 
first, MITRE evaluated the 12 member sites for compliance with MOU requirements 
(including an initial landing page analysis); second, MITRE evaluated the 12 member sites 
and the Free File-related pages on IRS.gov against recognized software user interface 
usability principles (called heuristics) in order to identify potential usability problem 
areas. For both parts, MITRE set-up user accounts and ran scenarios to test the sites in a 
systematic, exhaustive fashion. MITRE’s usability assessment used both industry- 
recognized criteria (heuristics) as well as MITRE’s interpretation of how MOU 
requirements translate into usability criteria and whether sites are optimized for mobile 
devices. 

• Taxpayer Usability Testing. MITRE conducted a study to test the usability of the IRS 
Free File portal and assess taxpayers’ actual experience in filing tax returns using three 
randomly selected Free File software offerings. MITRE conducted in-person usability 
testing sessions with 29 taxpayers who filed their taxes electronically using Free File 
software in prior year filings. During the testing, taxpayers were observed as they 
completed fictitious tax scenarios using the IRS Free File portal and Free File software. In 
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addition to data on the usability of the Free File services, MITRE collected and analyzed 
data on the frequency of errors committed, taxpayer satisfaction, and experience (via a 
brief questionnaire), and taxpayers’ perceptions of the Free File program. 

• Legal Analysis. MITRE’s General Counsel, in coordination with outside counsel, 
conducted a review of the MOU’s terms and conditions to determine recommended 
revisions to the MOU to more clearly define the parties' responsibilities to enhance and 
promote the IRS Free File program, address any shortcomings in the MOU language as it 
relates to the prohibition against unfair methods of competition, and provide enforcement 
capability via liability clauses if the MOU's terms and conditions are breached by a 
Free File member organization. 

• Recommendations. Based on the data gathered from the workstreams, MITRE developed 
recommendations for overall improvements to the Free File program, potential 
enhancements to the MOU, and improvements to usability ofIRS.gov and some aspects of 
member offerings. 

Figure ES-1 shows how each of these workstreams collectively contributed to addressing 
targeted assessment areas (assessment objectives) including: documenting the current Free File 
program environment, assessing Free File program management and oversight, assessing FFA 
member compliance with the MOU, assessing taxpayer experience with respect to the program, 
and ultimately drafting recommendations for the program and MOU. Outputs from some 
workstreams contributed to more than one assessment area. 



Document Current 
Free File Environment 


Z Trends 

Z Free File Market 
Z External Literature 



Assess Free File Program 
Management & Oversight 




Assess Free File Alliance (FFA) 
Member Compliance 




Taxpayer Experience 


Z Program Objectives 
V Program Management 
Z IRS Compliance Approach 
Z FFA Compliance Approach 


V IRS Data Analysis 
Z Web Analysis 
Z Taxpayer Journey 
Z FFA Leadership and 
Member Interviews 


Z Taxpayer Journey 
Z Taxpayer Usability Testing 



Develop Program & MOU 
Recommendations 


Z Legal Review 
Z SME Review 
Z IRS Interviews 
Z FFA Leadership and 
Member Interviews 
Z Workstream Findings 


Figure ES-1. High-Level Approach to Free File Program Assessment (Aligned to Assessment Objectives). 
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Addressing Public Perceptions 

Throughout the assessment, MITRE considered the perspectives of various media, oversight, and 
advisory organizations: the recent ProPublica articles on the Free File program, the 2018 Internal 
Revenue Service Advisory Council (IRSAC) Annual Report, and the 2018 National Taxpayer 
Advocate Annual Report to Congress 4 . Three primary themes emerged from these external 
sources: 

1. Member companies use deceptive strategies to steer taxpayers away from using the Free 
File program services towards fee-based services. 

2. The IRS Free File Program Office does not provide adequate oversight. 

3. Low usage rates signal that the Free File program is not working. 

However, MITRE’s findings from this assessment determined that while there may be an 
element of truth to each of the criticisms, the data indicate the issues are much more nuanced and 
don’t tell the full story. 

MITRE conducted the assessment of the Free File program in its current state—the tax software 
industry providing free tax preparation and filing services under a MOU with the IRS. It was not 
in the scope of MITRE’s effort to address the underlying political issue that is the genesis of 
much of the controversy surrounding the Free File program—whether the IRS should create its 
own free tax preparation and filing program or continue to rely on the tax software industry to 
provide these services. While MITRE recognizes that government services provided to its 
citizens are continually evolving, and that Congress may determine in the future that the IRS 
should offer that service (and provide it with the appropriate funding to do so), for the purposes 
of this study, MITRE assumes that industry will continue to be the entity that provides free tax 
return preparation and filing offerings to taxpayers. All of MITRE’s recommendations in this 
report are based on that assumption. 

Search Engine Avoidance 

The issue that seems to have created the most publicity and was at the center of much of the 
congressional interest is the practice of coding that caused members’ Free File landing pages to 
be excluded from an organic search through Google or Bing search engines. During FS 2019, 
five of the 12 FFA members did engage in a practice that excludes their company’s Free File 
landing page in organic searches. When questioned about it, most members contended this keeps 
them in compliance with the MOU (language in the MOU preamble) and in alignment with IRS 
marketing that states Free File is “accessible only through IRS.gov.” 5 MITRE did not take a 
position on the practice—determination of whether this is a deceptive business practice is not 
within MITRE’s legal purview. The perception points to the need for the IRS to take a position 
on the practice and incorporate new language into the MOU. MITRE also conducted an in-depth 
analysis of what the results of organic searches provided. 


4 Internal Revenue Service, "Internal Revenue Service Advisory Council Public Report, Publication 5316, Catalog Number 
71824A," Internal Revenue Service, Washington, D.C., 2018.; Taxpayer Advocate Service, "National Taxpayer Advocate Annual 
Report to Congress," Taxpayer Advocate Service, Washington, D.C., 2018.; Series of ProPublica articles from March 20, 2017 to 
June 5, 2019, available at https://www.propublica.org 

5 Internal Revenue Service, "Changes strengthen program; Free File can also help navigate new tax law provisions," Internal 
Revenue Service, 11 january 2019. [Online]. Available: https://www.irs.gov/newsroom/irs-free-file-opens-today-in-advance-of- 
tax-season. [Accessed 10 September 2019], 
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• Five of the 12 members used a coding device 6 to keep 
their Free File landing page out of organic searches. 


Member companies use deceptive 
strategies to steer taxpayers away from 
using the Free File program services 
towards paid commercial services 
(ProPublico). 


• Based on a representative sample of organic (unpaid) 


search keywords for free tax filing, the highest 
number of searches went to IRS.gov (>780,000). The 
top member landing page had 698,000 searches. The 
next highest was less than 180,000 searches. The 


total unpaid traffic to the 12 members was 1.8 
million. 

• Paid searches resulted in the highest amount of 
traffic (using the same representative sample of 
keywords as above), more than 10 million total to the 
12 member sites combined, none to IRS.gov (IRS does 
not pay for advertising). 

• IRS.gov Free File landing page received 2.8 million 
entrances 7 (the landing page was the first page a user 
saw when arriving on IRS.gov) during FS 2019. 

• Four of the five members who engaged in the 
practice had increases in Free Filing in FS 2019, and 
one remained relatively flat. Five of the seven who 
did not engage in the practice had increases in Free 
Filing, and two had decreases. 

• Number of taxpayers who received free filing outside 
of Free File through free commercial offerings is not 
publicly available. However, according to the FFA, the 
current members provided a total of more than 17 
million of these free commercial filings in FS 2019 8 . 


6 “<META NAME="ROBOTS" CONTENT="NOINDEX">meta 

7 Because the IRS does not employ any tracking codes once a user leaves the site, there is no way to know how many of these 2.8 
million entrances became Free File users. 

8 The actual number of free offerings may be higher. One company who was a member in 2019 did not provide a number. 
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Table ES-1 depicts the effects of organic versus paid searches for the top 108 non-branded high 
value keyword search terms that drove traffic to Free File member landing pages. This data was 
derived from website traffic information downloaded from S EM rush 1 ', a digital market software 
as a service company. 


Table ES-1. Comparing Paid Search Traffic to Organic Search Traffic 9 . 


FFI Member Sites 

(Member names redacted) 

Est Traffic to 

FFI Landing Pages 

Est Traffic to 

Non-FFI Landing Pages 

Est Paid Traffic to 
Non-FFI Landing Pages 

Est Cost 

for Paid Traffic 


0 

3,823 

246,856 

$416,912 


378 

248 

0 

0 


648 

65 

0 

0 


0 

11 

54,594 

$91,616 


0 

0 

0 

0 


3,370 

0 

0 

0 


0 

697,756 

5,400 

$103,270 


3,488 

95,688 

1,812,769 

$3,060,979 


205 

179,161 

3,662,118 

$6,213,910 


11,696 

562 

0 

0 


36,310 

9,547 

3,039,163 

$5,125,896 


0 

40,759 

1,446,217 

$2,426,933 

Total 

56,095 

1,812,218 

10,267,117 

$17,347,900 

IRS.gov (for comparison) 

700,780 

83,819 

0 

0 


The data in Table ES-1 shows that for the organic searches, more traffic went to the IRS.gov 
Free File landing page than to any one of the individual member sites’ non-Free File landing 
pages. However, the most traffic, as would be expected, went to paid advertising. Seven of the 
12 Free File members bought ads for keywords related to free tax filing that directed traffic to 
the landing pages for their non-Free File free tax filing software offering, resulting in five times 
more traffic than organic searches. The data used for this analysis (not shown in this table) 
indicated that no member had paid advertising for their Free File landing pages. This is not 
unexpected, as the MOU specifically assigns advertising responsibility to the IRS. However, 
during the FFA member interviews, one member stated that their company does employ 
advertising practices for the Free File program 9 10 . 

Program Oversight 

The perception that members were undertaking potentially deceptive practices that resulted in 
vulnerable taxpayers paying for tax services-compounded by the program’s low participation 
rates-led to criticism that the IRS Free File Program Office was not providing adequate oversight 
of member behavior. What this critique overlooks is the fact that the Free File program operates 
as a public-private partnership and, as such, program oversight is a mutual, collaborative effort 


9 SEMrush Inc., "All-in-one Marketing Toolkit," SEMrush Inc., 2019. [Online]. Available: https://www.semmsh.com/. [Accessed 
week of 27 May 2019], 

10 During MITRE interviews with representatives of FFA member organizations, one member indicated that their company pays 
for advertising of the Free File program through a number of methods. However, the interview did not include detailed discussion 
of advertising practices specific to key word search. For this analysis, MITRE looked specifically at ads purchased for key words 
related to free tax filing. The results of this analysis indicated that no purchased ads directed users to the landing pages of Free 
File offers. 

PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 

viii 








PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 


with the industry, represented by the Free File Alliance. While there is room for improvement, 
based on assessment findings, MITRE considers the IRS’s current compliance processes 
adequate and effective to support the integrity of the program. In part, because of this long¬ 
standing partnership with FFA and continuous assessment and improvement of the program as it 
has evolved. 



• Because the Free File program is a PPP, and not a 
paid contractual relationship, it must work 
collaboratively with industry on oversight of the 
program. 


The IRS Free File Program Office does 
not provide adequate oversight (IRSAC, 
Taxpayer Advocate Service [TAS], 
members of Congress). 


• IRS program office conducts member website 
reviews, evaluates member offer coverage and 
monitors member acceptance rates, while working 
closely with FFA to address, document, and resolve 
any identified or potential compliance breaches. 


• IRS's existing agency governance of the Free File 
program effectively allows the program to fulfill the 
IRS's currently articulated program goals. 


• While the program office does not have updated, 
measurable objectives for internal agency 
performance, its shared oversight of member 
compliance strictly follows the provisions of the MOU 
and meets joint program objectives. 


Taxpayer Participation in Free File Program 

Finally, what has led many to question the overall effectiveness of the Free File program is the 


participation numbers themselves. In 2018, fewer than 3 million filers used the Free File program 
to file their tax year 2017 returns out of nearly 104 million taxpayers that meet the eligibility 
requirements for the program. However, these numbers don’t tell the entire story and take the 
issues of taxpayer choice and taxpayer behavior out of the equation. The Free File program will 
generally appeal to those taxpayers that prefer a “do-it-yourself’ (DIY) method of tax 
preparation and filing. Looking retrospectively at the eligible population from FS 2018 and how 
those taxpayers chose to file can help the IRS better understand their target population for Free 


File. 
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What's been said... What the data show... 


Low usage rates signal that the Free 
File program is not working (TAS, 
ProPublica, members of Congress). 


• While 103.8 million taxpayers were technically 
eligible to use Free File in FS2018, half of those used a 
paid preparer, taking them out of the do-it-yourself 
(DIY) tax preparation pool in that year. 

• Another 9.8 million taxpayers in the DIY pool received 
refund anticipation checks (RACs) or refund 
anticipation loans (RALs), taking them out of the 
eligible pool. 

• Though paper filers are a potential target population 
for Free File, the 8.8 million DIY taxpayers who filed 
on paper removed themselves from the 2018 
candidate pool. 

• Three million taxpayers went to Volunteer Income 
Tax Assistance (VITA) sites, removing them from the 
candidate pool. 

• The actual number of filers using DIY software and 
eligible to use Free File in FS 2018 was about 30 
million (see Figure ES-2). Of those, about 9 percent 
used Free File. 

• Of those 30 million, a significant percentage (17.7 
million for FS 2019 according to FFA) received free 
commercial filing from the current members. 

• The VITA program (the bricks and mortar assisted 
free equivalent of Free File) had approximately 3 
million taxpayers use their service VITA has 
thousands of volunteer preparers, and more than 
2,000 partner organizations who conduct outreach 
for their 11,000 sites nationwide. 


As shown in Figure ES-2, once the number of taxpayers that choose to file via a different 
method—e.g., with a paid preparer or through a Volunteer Income Tax Assistance (VITA)—or 
elects to receive their tax refund immediately through a refund anticipation loan (RAL) or a 
refund anticipation check (RAC)—are removed due to their preferences, the actual pool of 
candidates for the Free File program from 2018 is about 30 million taxpayers. A significant 
percentage of those receive free filing outside the Free File program. Though the numbers for 
free commercial filings are unknown for tax year (TY) 2017, a study by the FFA in 2015 showed 
it to be 19 million for that year from the 14 members, and the FFA attested to 17.7 million for 
TY 2018 from 11 current members. 
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103,778,000 Eligibile Taxpayers - TY 2017 



■ Paid Preparers 

■ RAC/RAL self-prepared 

■ Paper filers - self-prepared 

■ VITA/TCE 

■ Free Filers 

■ Potential Add'l Free Filers 


Figure ES-2. Break-down of 103,778,000 Taxpayer Eligible for Free File (TY2017). 

It is outside the scope of this assessment for MITRE to determine what the proper percentage of 
Free File taxpayers should or could be. Without further analysis and research into taxpayer 
behavior it is impractical to make that type of determination. Presumably if awareness was at 
peak saturation, participation would be greater than it is now. However, there are many 
competing free offers in the market that even taxpayers who are aware of Free File may gravitate 
to. The IRS’s ability to affect the Free File members’ behavior in terms of their commercial 
offerings, or taxpayers’ behavior in terms of how they choose to file their return, is limited. What 
the IRS does have control over is what and how much they do to ensure program awareness. 

Free File Member Assessment Results - MOU Compliance 

Recent attention and criticisms notwithstanding, the Free File program has operated for 17 years 
under the governance of an MOU—an agreement that all members must abide by or be removed 
from the program. The IRS asked MITRE to conduct a fact-finding review of each member to 
determine whether members were in compliance with the most recent version of the MOU at the 
time of the assessment. 

To assess MOU compliance, MITRE drew findings from IRS interviews and document reviews, 
interviews with Alliance leadership and an Alliance-contracted reviewer, and fact-finding 
interviews with representatives from each of the Free File member organizations. In addition, 
MITRE incorporated the results of the hands-on taxpayer journey reviews of member Free File 
landing pages and websites, as well as results from the web forensics analyses where applicable. 

Overall, based on the various assessment workstreams, all Free File members were compliant 
with the MOU, in particular with Article 4, Standards of Practice. 

PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 


xi 














PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 


Article 4 outlines level of service, software and website functionality, security, and disclosure of 
forms and schedules. In addition, it includes taxpayer service options and other requirements 
such as innovation. MITRE found two instances of MOU non-compliance and an inconclusive 
finding for nine requirements for some or all members. There were also two requirements that 
were not applicable at the time of review for several members. 

The two non-compliance instances were: 

• One member was non-compliant with a sentence in MOU 4.15.4. The MOU states that 
“Members must clearly list their free customer options ... on the Member’s Free File 
Landing Page (or such page must have a clear and prominent link to such disclosures 
directly from this page).” Based on MITRE’s landing page analysis, the link was not 
“clear and prominent” on their landing page 11 . 

• One member was non-compliant with MOU 4.32.4. The MOU requires members to 
“communicate not less than once annually via email with their taxpayer customers who 
used Free File services and completed their returns through Free File in the immediately 
preceding tax year.” The representative from the member acknowledged they were 
supposed to initiate an email to their prior tax season Free File taxpayers and self-attested 
that they did not provide the required email prior to the tax season. 

While MITRE deems these to be non-compliant issues, none rise to the level of severity or 
concern regarding the integrity of the Free File program. 

Taxpayer Experience Analysis 

In addition to assessing compliance with MOU requirements, MITRE also conducted research 
activities to understand the taxpayer experience using the Free File program. These research 
activities were designed to represent the perspective of the end user of the Free File system—the 
taxpayer. As such, MITRE identified pain points and areas for improvement to make the 
experience easier, clearer, and more accurate for taxpayers. This included an expert review of 
IRS.gov and the 12 member sites for potential usability issues, and usability testing interviews 
with actual taxpayers. 

Taxpayer Journey Analysis 

MITRE conducted a taxpayer journey analysis to evaluate potential usability issues with the Free 
File member systems and Free File-related web pages. The taxpayer journey analysis also 
evaluated member landing pages for compliance with the MOU, as described above. The rest of 
the analysis was focused on the taxpayer experience from a usability perspective and should not 
be interpreted as evidence for or against member compliance with the MOU. 

The taxpayer journey analysis used both established industry criteria and MOU requirements 
related to system usability and user expectations for system functionality as anchor points for the 
analysis. Specifically, MITRE used the following sets of criteria to conduct the review of the 12 
member sites and IRS.gov’s Free File-related web pages: 


11 As of the date of this report, the member has updated their Free File landing page. 
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• Nielsen’s 10 usability heuristics 12 . These 10 criteria, called “heuristics,” are widely 
recognized measures of system usability. Taken together, the 10 heuristics construct a 
measure of user experience and include factors such as “match between system and the 
real world,” “error prevention,” “flexibility and ease of use,” and “consistency and 
standards.” 

• MOU requirements, from a usability perspective (user experience [UX] MOU 
Adherence). MITRE examined MOU requirements pertaining to the user experience and 
constructed usability criteria from these MOU requirements. These criteria assess user 
interface elements closely associated with MOU requirements and included factors such as 
“understands offerings available” and “transparency around ineligibility.” MITRE’s 
assessment against these criteria are not included in the evaluation of MOU compliance 
presented in Section 4.0. 

• Mobile experience. MITRE evaluated the extent to which the examined sites were 
optimized for mobile device use. 

For all three of these groups of criteria, the objective of the evaluation was to identify potential 
usability issues and prioritize them via a 1-5 (1 = worst, 5 = best) severity score. 

MITRE’s analysis found that, while the Free File members adhere to many UX best practices as 
measured by Nielsen’s usability heuristics, they could improve the most in “error prevention” 
and “match between system and real world” heuristics. With respect to examining the sites in 
tenns of a usability interpretation of MOU requirements, the area where members have the most 
need for improvement was “transparency around ineligibility.” 

All of the scores from the review are rolled up into a dashboard that shows a quick view of the 
summary scores for user experience, UX MOU adherence, and mobile experience, with a more 
detailed breakout of the scores from the heuristic review by each heuristic and grouping of MOU 
requirements (see Section 5.1). Figure ES-3 shows the summary dashboard from the taxpayer 
journey heuristic analysis, and aggregates Nielsen’s 10 heuristics under “user experience” issues 
related to the usability in the MOU under “UX MOU adherence,” and whether or not the site is 
mobile-optimized under “mobile experience.” The 12 member sites are shown in the columns by 
number and the heuristic dimensions are displayed down the rows. 


12 Nielsen’s 10 Usability Heuristics: Nielsen, Jakob. (1994). Heuristic evaluation. In Nielsen, J., and Mack, R.L. (Eds.), Usability 
Inspection Methods, John Wiley & Sons, New York, NY 


PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 



PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 


Dimension 


User Experience 


UX MOU Adherence 


Mobile Experience 


DDDBDDB 





♦ ♦ t # ♦ t 4 
******* 
******* 


8 9 10 11 12 


* * * * * 

* * * * * 

* * * * * 


SUMMARY 

At a glance, the Free File providers adhere 
to many UX best practices. Overall, they 
could improve the most in “Error 
Prevention” and “Match Between System & 
Real World” heuristics. The most significant 
UX MOU improvement to make across the 
board is “Transparency around ineligibility.” 


Score range 


85% to 100%. Area of strength. 

^ 70% to 84%. Room for improvement. 

0% to 69%. Area of weakness. 


Figure ES-3. Heuristic Evaluation Dashboard Summary. 

The top two highest-rated members both had exceptional user experience, intuitive mobile 
experiences, and followed the MOU’s guidance related to taxpayer experience and usability. 
Other members might consider mirroring the transparency Member 10 has around Free File 
eligibility, which lets the user know quickly whether or not they are eligible to proceed through 
the Free File process. 

Two members received the lowest ratings for the usability of their user interfaces. Three 
members did not have mobile-optimized sites, which would make it very difficult for users to file 
their taxes from a mobile device. 

Taxpayer Usability Analysis 

MITRE conducted the taxpayer usability study via in-person interviews with 29 taxpayers who 
previously used Free File from selected zip codes in Chicago. To select an interview location, 
MITRE reviewed the density of TY 2018 Free Filers in multiple clusters of zip codes in U.S. 
metropolitan areas; Chicago had a relatively high volume of Free Filers, which allowed MITRE 
to send invitation letters to potential participants and elicit a sufficient interview sample. MITRE 
randomly selected a subset of Chicago Free Filers to receive invitation letters to participate. 

The objectives of these interviews were to understand how taxpayers locate the Free File 
program options; understand how taxpayers select a software provider; capture usability issues 
with IRS websites related to the Free File program; capture usability issues with specific vendor 
platforms; and understand taxpayers’ previous experiences with the Free File program. During 
the sessions, participants completed fictitious scenarios to evaluate the experiences of completing 
three tasks: 1) locating the Free File program online; 2) selecting a Free File software offer, 
including with the software lookup tool; and 3) completing a tax return. MITRE randomly 
selected three Free File member software offers to test with taxpayers. Each participant filed a 
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fictitious return with one of the three selected offers. Taxpayers used mocked-up information 
(e.g., fictitious Form W-2s and other documentation) to complete the tasks. 

MITRE identified and reported all usability issues observed by the research team with each of 
the three Free File usability tasks and provided suggested solutions for addressing each of these 
issues individually. Drawing on the data collected from the usability tasks, analysis of qualitative 
data gathered during debriefing and with semi-structured interview questions, the satisfaction 
and user experience questionnaires, and metrics on taxpayer behaviors and selections during the 
tasks, MITRE identified the several high-level findings. When it comes to navigating to the Free 
File program and selecting a member offer, taxpayers: 

• Lacked understanding of what the Free File program is and its relationship to the IRS 

• Struggled to find the Free File landing page, including through IRS.gov 

• Experienced difficulties and confusion with selecting a software offering, particularly 
when evaluating Earned Income Tax Credit (EITC) eligibility 

• Expressed that they were overwhelmed with choices and information 

• Used strategies beyond eligibility criteria to select software offers 

With respect to using member-specific Free File program tax preparation software, common 
usability issues included trouble finding where to report adjustments to income, difficulty 
choosing between filing statuses, and trouble navigating different systems’ “flow.” 

Overall, taxpayer ratings of ease and satisfaction varied by task, vendor, and taxpayer 
demographic characteristics. 

Findings and Recommendations 

Building upon the analysis from the various workstreams in this assessment, MITRE provides 
the following findings and recommendations for improvements to the Free File program and to 
the Free File MOU that governs the program. The findings and recommendations are grouped by 
key themes that emerged through the execution of the MITRE assessment. The recommendations 
include actions that the IRS may take to improve the integrity and effectiveness of the Free File 
program. 

Where appropriate, MITRE provides workstream-specific recommendations the body of this 
assessment. 
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1. Objectives of the Free File program have not been updated since the 
program’s inception. 

Findings 

Recommendations 

The IRS and FFA have not revisited the 
objectives of the program since its 
inception. The objectives (reflected in the 
MOU) are outdated and don’t reflect 
leadership’s current view of program 
success. 

The program objective of providing the 
venue for free tax filing for 70 percent of 
the population has been met. The e-filing 
objective has been met. No other metrics 
currently exist by which the program office 
can measure its performance. 

MITRE recommends that IRS leadership, in 
collaboration with the FFA, define new program 
objectives and create metrics to determine the 
Program Office’s performance against those 
objectives. Recognizing that the IRS and FFA 
will need to negotiate any formal changes to 
joint program objectives, MITRE recommends 
that IRS develop internal objectives and metrics 
that reflect its own priorities. While agreement 
between the IRS and FFA would be the ideal, 
translating priorities into objectives and metrics 
should not be contingent on that agreement. 

For example, according to IRS leadership, its 
focus for free file is less about target number of 
participants, and more about maximizing 
awareness to ensure people recognize it is a 
choice. 

Potential metrics could include: 

• Number of companies participating, 
potentially targeting a specific market 
share 

• Actual number of Free File taxpayer 
candidate pool/participants (e.g., xx% of 
the DIY community minus RALs/RACs) 

• Metrics associated with compliance (e.g., 
offering parameters, etc.) 

• Awareness numbers (see additional 
recommendations) 

• Web analysis metrics (e.g., number of 
clicks on the Free File link on the 

IRS.gov home page, percentage of select 
keyword searches that bring taxpayers to 
the IRS.gov landing page, etc.) 
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2. If Free File objectives are revised to focus on taxpayer awareness, more 
needs to be known about the target population and taxpayer behavior. 

Findings 

Recommendations 

The IRS has collected demographics data 
on users of Free File and migration data of 
those users, but they do not have 
demographics data of the potential pool of 
Free File users. This data would be helpful 
in targeting outreach, (see Section 3.3) 

MITRE recommends the IRS conduct data 
analysis of the demographics of the population 
who are prime candidates for Free File but not 
using it. 

The total population pool of 103 million 
taxpayers who are eligible may not 
represent an accurate picture of the pool of 
taxpayers who are candidates for Free File. 

Taxpayers make choices that take them out 
of the candidate pool. About 9 percent of 
taxpayers who have not taken themselves 
out of the candidate pool by (for example) 
using a paid preparer or choosing a RAC or 
RAL, use Free File (not taking into 
consideration those who choose free 
commercial offerings) There is no 
consensus on what would be an acceptable 
number. 

MITRE recommends the IRS conduct a taxpayer 
behavior study to better understand the factors 
involved in a taxpayer’s choice of filing 
methods. Understanding taxpayer motivation 
and choices could help determine what would be 
an acceptable Free File number, if a number is 
needed at all. This aligns with the IRS-related 
priorities of the internal MITRE research 
program and will be proposed for a future 
research project. 

While each individual member receives 
customer feedback in various ways for 
their own Free File product, the IRS has 
not conducted a customer survey since 

2009. 

MITRE recommends the IRS conduct a 
customer survey of Free File users specifically to 
the IRS experience of Free File (not the software 
they filed their return through) to use as a 
baseline and create an online survey for users to 
capture continuous feedback. 

The IRS has no access to data that would 
indicate whether a DIY filer paid for a 
return or received a free commercial 
offering. As the issue is whether a low- 
income taxpayer has to pay for filing, 
knowing how many taxpayers receive free 
filings either through Free File or a 
commercial product would give the IRS a 
better understanding of their target 
markets. 

MITRE recommends, that in the spirit of 
transparency, the IRS propose the industry to use 
an indicator on free commercial filings 

Or 

MITRE recommends the IRS request that FFA 
provide them the aggregate number of taxpayers 
who file their taxes using free commercial 
products each year. When developing metrics, 
this could help with understanding a more 
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accurate picture of the population of the 

Free File candidate pool. 

Leadership acknowledged that the number 
of returns submitted through Free File have 
not increased in the same way that e-file 
numbers have and articulated a goal of 
increased awareness. The last awareness 
study was conducted in 2011. 

MITRE recommends the IRS commission an 
awareness study to baseline taxpayer awareness 
of Free File, specifically of targeted 
demographics detennined in the prior 
recommendation. 

MITRE further recommends, based on the 
results of this study—and other recommended 
research—the IRS conduct a targeted awareness 
campaign to the desired Free File demographics, 
and include other IRS-sponsored venues such as 
low-income tax clinics (LITCs) and VITA sites, 
including IRS’ own website. 

Paid advertising yields results. In MITRE’s 
web analysis of those searching for free tax 
filing, an estimated 85 percent of the traffic 
followed a paid search result to member 
sites, while the other 15 percent of visits to 
member sites came from an organic search 
result. Searchers were more than five times 
likely to click on a paid search result. 

MITRE recommends the IRS conduct a cost- 
benefit analysis to determine whether to 
purchase paid advertising to reach taxpayers 
searching for free tax filing. We recognize the 

IRS may have to seek statutory authority and 
budget-specific funding for such advertising if it 
was deemed beneficial. 


3. IRS Free File program is a long-standing public-private partnership, but 
there is a lack of public awareness about what that means to the program. 

Findings 

Recommendations 

Free File program is a partnership between 
the IRS and industry that requires 
cooperation to achieve success. This 
partnership requires a balance that serves 
the interests of the government and 
taxpayers, but also creates a value to the 
for-profit industry that provides the 
service. 

The IRS Free File program could benefit 
from providing clarity and scope around 
the PPP, both internally and externally. 

The purpose and scope of PPPs are not 
common, and awareness is key to success. 

MITRE recommends that a health assessment of 
the PPP become part of the overall metrics for 
the Free File program, using a generally 
accepted framework provided by external 
sources, such as that used by MITRE in this 
assessment. 

MITRE further recommends that the IRS 

Free File Program Office incorporate talking 
points addressing the unique nature of the PPP in 
tenns of joint governance and the value a PPP 
provides to all partners as central themes when 
addressing critics or publicly discussing the 
program to help address this lack of awareness. 


PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 

xviii 





PRE-DECISIONAL - FOR INTERNAL IRS W&l USE ONLY - NOT FOR PUBLIC RELEASE 


4. Rules that were put into the MOU to promote fairness among FFA 
members and promote choices for taxpayers translate into unclear 
presentation of offers and confusion for taxpayers. 

Findings 

Recommendations 

The FFA is obligated to provide Free File 
to eligible taxpayers through individual 
commercial sites such that, when taken in 
aggregate these services are offered to the 
lowest 70 percent of the taxpayer 
population, calculated using AGI. 

Moreover, the MOU requires that each 
company must offer services to at least 10 
percent of total eligible taxpayers but to no 
more than 50 percent of taxpayers (“10/50 
rule”)—a requirement set to level the 
playing field for smaller providers. 

While the 10/50 rule has been (and may 
continue to be) beneficial to FFA members 
to ensure equity, it’s possible that it has 
outlived its usefulness. MITRE did not 
have adequate time, nor was it within 
scope, to do an analysis of how the 10/50 
rule impacts Free Filed returns. However, 
from a taxpayer perspective, the different 
company offerings create confusion as a 
taxpayer navigates the various choices. 

MITRE recommends the IRS conduct an 
analysis/study of the 10/50 rule. This could 
include a confidential market study of the 
members to help understand the economic 
incentive to assist in preserving competition 
amongst the members and promoting taxpayer 
economic welfare; a small pilot to detennine the 
business impact of removing the rule; a 
consideration of whether differing rules for the 
two large companies would benefit or create 
inequity; and/or an in-depth data analysis of how 
the different offerings have impacted members’ 
free file numbers. 

Even if the rule is not changed, MITRE 
recommends simplifying the display of 
members’ offering to reduce taxpayer confusion. 


5. Usability activities—both with taxpayers and by usability experts— 
provide insight into how taxpayers experience various aspects of the 

Free File program. 

Findings 

Recommendations 

MITRE conducted a complete heuristic 
assessment of the Free File website and 
members websites, and a usability 
assessment with Free File users. Both 
assessments found that improvements 

A complete list of recommendations to improve 
usability and the user experience on IRS.gov is 
provided in the Taxpayer Experience Results 
Report (Appendix G) and taxpayer journey 
supplemental materials (Appendix F). While 
usability-specific, these recommendations 
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could be made to enhance the user complement the program-level recommendations 

experience. above and as such may be repetitive. 

MITRE recommends that IRS: 

• Standardize how Free File is referenced on 
IRS platfonns and communications 

• Take steps to increase ease of understanding 
eligibility requirements 

• Ensure that taxpayers understand that 
determining eligibility for a software offer is 
ultimately their responsibility 

• Recommend best practices to increase 
usability of members’ tax preparation 
software systems 

• Conduct research on the usability taxpayer 
access and use of the Free File system 

• Conduct research on Free File awareness, 
attending to different demographic groups’ 
awareness 

• Eliminate or change the EITC eligibility 
question in the lookup tool and on the 
software offers page 

o If the IRS stated on the Free File site that 
all taxpayers eligible for EITC are 
eligible for Free File, the question could 
be removed from the tool. MITRE 
recognizes this would be a topic for 
negotiation between the IRS and FFA 
because of how that could impact some 
companies’ appearance in the tool. 

o If the IRS chooses to leave EITC in the 
lookup, MITRE recommends linking the 
question to the EITC eligibility tool, but 
ONLY after conducting usability testing 
on the EITC eligibility tool and 
addressing the results. 
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6. The MOU is an evolving document that is added to with each new 
negotiation, resulting in a piecemeal agreement that is out of date in some 
areas. 

Findings 

Recommendations 

The MOU generally has provisions added 
with each revision, but few are taken out. 
Much of the same language is used, not 
only in the objectives, but also in the 
technology portion, that has become 
outdated (e.g., text CAPTCHA). 

MITRE recommends that the IRS and the 

Alliance jointly detennine new, measurable 
objectives for the Free File program, and draft a 
new MOU, pulling and using provisions from 
the current MOU that are relevant, discarding 
provisions that are not, and creating new 
provisions based on the current environment. 

All but two of the members’ Free File 
offerings are the same as their commercial 
offerings. The security/privacy rules of 

Free File are the same as e-File, and all 

Free File members are also members of the 
Security Summit where the IRS works 
collaboratively with the industry to 
improve security. 

MITRE recommends simplifying the MOU in its 
security and privacy provisions by simply 
requiring members to adhere to the e- 
File/Security Summit requirements and 
recommendations. 

MITRE’s review of the individual 
members’ processes for applying the 

Free File indicator (without looking at the 
actual coding) indicated most members 
used the same process that should place the 
indicator only on Free Filed returns. 
However, that is not always the case. 

In MITRE’s data analysis of returns 
marked with the Free File indicator, 
several thousand returns from one vendor 
indicated the presence of a RAC. The 
vendor noted that the IRS infonned them 
of this irregularity, and they discovered a 
coding error had mistakenly placed the 
indicator on returns that had not been Free 
Filed, which was subsequently corrected. 
Another vendor also had several hundred 
returns with the Free File indicator on 
returns with AGI of more than $66,000. 

MITRE recommends adding language to the 

MOU to specify conditions under which the 
indicator is applied to ensure quality control. 

Though not necessarily dependent on the MOU, 
MITRE also recommends the IRS develop a 
quality control method that would allow the IRS 
to validate returns with the Free File indicator 
were actually Free Filed. 
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7. Search exclusion techniques are the latest new issue for consideration 
when entering into the next round of MOU negotiations. 

Findings 

Recommendations 

Five of the companies used a NOINDEX 
code to exclude their company’s Free File 
landing page from organic searches. They 
reported that their belief was that this kept 
them compliant with the MOU. However, 
while the MOU states the IRS will provide 
links to their site, it does not prohibit 
taxpayers from accessing their sites 
directly. IRS press releases state that 
taxpayers can ONLY access Free File sites 
through IRS.gov. 

MITRE recommends that if the IRS wants 
taxpayers to go to Free File sites only through 
IRS.gov, the MOU should reflect that 
specifically as a provision, as opposed to the 
current reference in the preamble. 

MITRE further recommends the IRS detennine 
their position on the use of NOINDEX and add a 
provision to the MOU addressing that use. 
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